Wittenberg University - Master of Science in Analytics
ANLT 510 - Advanced Statistics and Modeling
Day 1: Getting Comfortable with Uncertainty
05 Sep 2021
Supervised Learning Example (Cont.)
Visualizing loss functions
For our perfect example data, we can choose among several different loss functions
However, not all of them will return an accurate solution
In the sections below I walk through the choice of several different loss functions and plot the results
- A naive loss function as the difference between the observed output and the output returned by the proposed model
- A good loss function
- A better loss function
A Naive loss function
\[
Loss_{_{naive}}(\mathbf{y},\mathbf{x},m,b) = \sum_{i=1}^N y_i-m\times x_i-b.
\]
Using this function, loss would simply be defined as the sum of the vertical distances between each observed output \(y_i, i = 1,...,n\) and the output returned by the chosen model
The parameters \(m\) and \(b\) for the best-fit line correspond to model that has the minimum loss.
For our “ideal” data, the points fall on a straight line and we would expect the loss value in this case to be zero
Thus far, we’ve chosen a functional form that we believe is a good representation of the data – and a corresponding loss function
In the chunk below we define our naive loss function
loss_naive <- function(params,x,y) {
if(length(params) != 2) stop("Params should be a length 2 vector")
m <- params[1]
b <- params[2]
return(sum(y - m * x - b))
}
- Now, let’s use the
stats::optim() function to find the values of \(m\) and \(b\) that minimize the loss function and result in a model that best-fits the data
optim(par = c(1,1),
fn = loss_naive,
x = df$x,
y = df$y,
control = list(fnscale = 1))
$par
[1] 2.684172e+55 4.996335e+54
$value
[1] -1.111779e+57
$counts
function gradient
501 NA
$convergence
[1] 1
$message
NULL
A good loss function
- Another possible loss function would involve taking the absolute value of the errors – we know this function is minimized at zero
\[
Loss_{_{absolute}}(\mathbf{y},\mathbf{x},m,b) = \sum_{i=1}^N \Big\vert y_i-m\times x_i-b\Big\vert.
\]
- Once again let’s define our OK loss function
# First define a function to optimize
loss_absolute <- function(params,x,y) {
if(length(params) != 2) stop("Params should be a length 2 vector")
m <- params[1]
b <- params[2]
return(sum(abs(y - m * x - b)))
}
- Now, let’s use the
stats::optim() function to find the values of \(m\) and \(b\) that minimize the loss function and result in a model that best-fits the data
optim(par = c(1,1), # provide starting values for m and b
fn = loss_absolute, # define function to optimize
x = df$x, # provide values for known parameters
y = df$y, # provide values for known parameters
control = list(fnscale = 1))
$par
[1] 5 3
$value
[1] 3.301596e-06
$counts
function gradient
121 NA
$convergence
[1] 0
$message
NULL
- Let’s visualize this loss function – like we did before
A better loss function
The problem is that linear functions are unconstrained
A better option would be to propose a loss functions that is convex, such as
\[
Loss_{_{convex}}(\mathbf{y},\mathbf{x},m,b) = \sum_{i=1}^N \Big( y_i-m\times x_i-b\Big)^2.
\]
Note that we are minimizing the squared distances between the observed values and the proposed model - hence this is called least squares optimization
For the last time let’s define our convex loss function
loss_convex <- function(params,x,y) {
if(length(params) != 2) stop("Params should be a length 2 vector")
m <- params[1]
b <- params[2]
return(sum((y - m * x - b) ^ 2))
}
- Now, let’s use the
stats::optim() function to find the values of \(m\) and \(b\) that minimize the loss function and result in a model that best-fits the data
optim(par = c(1,1), # provide starting values for m and b
fn = loss_convex, # define function to optimize
x = df$x, # provide values for known parameters
y = df$y, # provide values for known parameters
control = list(fnscale = 1))
$par
[1] 4.999800 3.000373
$value
[1] 2.501828e-06
$counts
function gradient
71 NA
$convergence
[1] 0
$message
NULL
- Let’s visualize this loss function – like we did for the last two
How can I know which algorithm to choose?
- Model selection is part art and part science
